Systems and methods for onscreen menus in a teleoperational medical system

ABSTRACT

A method comprises displaying an image of a field of view of a surgical environment. A first medical instrument in the field of view may be coupled to a first manipulator in a teleoperational assembly. The method may comprise displaying a menu proximate to an image of the first medical instrument in the image of the field of view. The menu may include a plurality of icons wherein each icon is associated with a function for the first medical instrument. The method may also comprise identifying a selected icon from the plurality of icons based upon a movement of an operator control device of a teleoperational operator control system.

RELATED APPLICATIONS

This patent application is the divisional of U.S. patent applicationSer. No. 16/316,981, filed Jan. 10, 2019 which is the U.S. nationalphase of International Application No. PCT/US2017/029380, filed Apr. 25,2017, which designated the U.S. and claims priority to and the benefitof the filing date of U.S. Provisional Patent Application 62/362,376entitled “SYSTEMS AND METHODS FOR ONSCREEN MENUS IN A TELEOPERATIONALMEDICAL SYSTEM,” filed Jul. 14, 2016, which are incorporated byreference in their entirety.

FIELD

The present disclosure is directed to systems and methods for performinga teleoperational medical procedure and more particularly to systems andmethods for displaying function menus for teleoperational instrumentsused in a surgical environment.

BACKGROUND

Minimally invasive medical techniques are intended to reduce the amountof tissue that is damaged during invasive medical procedures, therebyreducing patient recovery time, discomfort, and harmful side effects.Such minimally invasive techniques may be performed through naturalorifices in a patient anatomy or through one or more surgical incisions.Through these natural orifices or incisions, clinicians may insertmedical tools to reach a target tissue location. Minimally invasivemedical tools include instruments such as therapeutic instruments,diagnostic instruments, and surgical instruments. Minimally invasivemedical tools may also include imaging instruments such as endoscopicinstruments. Imaging instruments provide a user with a field of viewwithin the patient anatomy. Some minimally invasive medical tools andimaging instruments may be teleoperated or otherwise computer-assisted.As teleoperational medical systems become more complex, with additionalfeatures and interaction modalities, adding additional physical controldevices to the operator control station becomes less feasible. To extendthe capability of a teleoperational system, graphical menus visible toand accessible by the operator are needed.

SUMMARY

The embodiments of the invention are summarized by the claims thatfollow below.

In one embodiment, a system comprises a teleoperational assemblyincluding an operator control system and a first teleoperationalmanipulator configured for operation by an operator control device ofthe operator control system. The first teleoperational manipulator isconfigured to control the operation of a first medical instrument in asurgical environment. The system also comprises a processing unitincluding one or more processors. The processing unit is configured todisplay an image of a field of view of the surgical environment anddisplay a menu proximate to an image of the first medical instrument inthe image of the field of view. The menu includes at least one iconrepresenting a function for the first medical instrument.

In another embodiment, a method comprises displaying an image of a fieldof view of a surgical environment. A first medical instrument in thefield of view is coupled to a first manipulator in a teleoperationalassembly. The method further comprises displaying a menu proximate to animage of the first medical instrument in the image of the field of view.The menu includes a plurality of icons wherein each icon is associatedwith a function for the first medical instrument. The method furtherincludes identifying a selected icon from the plurality of icons basedupon a movement of an operator control device of a teleoperationaloperator control system.

BRIEF DESCRIPTIONS OF THE DRAWINGS

Aspects of the present disclosure are best understood from the followingdetailed description when read with the accompanying figures. It isemphasized that, in accordance with the standard practice in theindustry, various features are not drawn to scale. In fact, thedimensions of the various features may be arbitrarily increased orreduced for clarity of discussion. In addition, the present disclosuremay repeat reference numerals and/or letters in the various examples.This repetition is for the purpose of simplicity and clarity and doesnot in itself dictate a relationship between the various embodimentsand/or configurations discussed.

FIG. 1A is a schematic view of a teleoperational medical system, inaccordance with an embodiment of the present disclosure.

FIG. 1B is a perspective view of a patient side cart, according to oneexample of principles described herein.

FIG. 1C is a perspective view of a surgeon's control console for ateleoperational medical system, in accordance with many embodiments.

FIG. 2 illustrates an input control device of the surgeon's controlconsole.

FIG. 3 illustrates a field of view of a surgical environment with agraphical menu proximate to a medical instrument in the field of view.

FIG. 4 illustrates a minimized graphical menu.

FIG. 5 illustrates a method for accessing features for a medicalinstrument.

FIG. 6 illustrates an opened graphical menu.

FIG. 7 illustrates another graphical menu.

FIGS. 8A-8C illustrate grip icons according to embodiments of thisdisclosure.

FIG. 9A illustrates an icon associated with an active state of afunction.

FIG. 9B illustrates an icon associated with a selected state of afunction.

FIG. 9C illustrates an icon associated with an enabled state of afunction.

FIG. 9D illustrates an icon associated with an unavailable function.

FIG. 10 illustrates a graphical user interface according to anotherembodiment of this disclosure.

FIG. 11 illustrates a graphical menu according to another embodiment ofthis disclosure.

DETAILED DESCRIPTION

For the purposes of promoting an understanding of the principles of thepresent disclosure, reference will now be made to the embodimentsillustrated in the drawings, and specific language will be used todescribe the same. It will nevertheless be understood that no limitationof the scope of the disclosure is intended. In the following detaileddescription of the aspects of the invention, numerous specific detailsare set forth in order to provide a thorough understanding of thedisclosed embodiments. However, it will be obvious to one skilled in theart that the embodiments of this disclosure may be practiced withoutthese specific details. In other instances well known methods,procedures, components, and circuits have not been described in detailso as not to unnecessarily obscure aspects of the embodiments of theinvention.

Any alterations and further modifications to the described devices,instruments, methods, and any further application of the principles ofthe present disclosure are fully contemplated as would normally occur toone skilled in the art to which the disclosure relates. In particular,it is fully contemplated that the features, components, and/or stepsdescribed with respect to one embodiment may be combined with thefeatures, components, and/or steps described with respect to otherembodiments of the present disclosure. In addition, dimensions providedherein are for specific examples and it is contemplated that differentsizes, dimensions, and/or ratios may be utilized to implement theconcepts of the present disclosure. To avoid needless descriptiverepetition, one or more components or actions described in accordancewith one illustrative embodiment can be used or omitted as applicablefrom other illustrative embodiments. For the sake of brevity, thenumerous iterations of these combinations will not be describedseparately. For simplicity, in some instances the same reference numbersare used throughout the drawings to refer to the same or like parts.

The embodiments below will describe various instruments and portions ofinstruments in terms of their state in three-dimensional space. As usedherein, the term “position” refers to the location of an object or aportion of an object in a three-dimensional space (e.g., three degreesof translational freedom along Cartesian X, Y, Z coordinates). As usedherein, the term “orientation” refers to the rotational placement of anobject or a portion of an object (three degrees of rotationalfreedom—e.g., roll, pitch, and yaw). As used herein, the term “pose”refers to the position of an object or a portion of an object in atleast one degree of translational freedom and to the orientation of thatobject or portion of the object in at least one degree of rotationalfreedom (up to six total degrees of freedom). As used herein, the term“shape” refers to a set of poses, positions, or orientations measuredalong an object.

Referring to FIG. 1A of the drawings, a teleoperational medical systemfor use in, for example, medical procedures including diagnostic,therapeutic, or surgical procedures, is generally indicated by thereference numeral 10. As will be described, the teleoperational medicalsystems of this disclosure are under the teleoperational control of asurgeon. In alternative embodiments, a teleoperational medical systemmay be under the partial control of a computer programmed to perform theprocedure or sub-procedure. In still other alternative embodiments, afully automated medical system, under the full control of a computerprogrammed to perform the procedure or sub-procedure, may be used toperform procedures or sub-procedures. As shown in FIG. 1A, theteleoperational medical system 10 generally includes a teleoperationalassembly 12 mounted to or near an operating table O on which a patient Pis positioned. The teleoperational assembly 12 may be referred to as apatient side cart. A medical instrument system 14 (also called “medicalinstrument 14”) and an endoscopic imaging system 15 (also called“endoscope 15”) are operably coupled to the teleoperational assembly 12.An operator input system 16 allows a surgeon or other type of clinicianS to view images of or representing the surgical site and to control theoperation of the medical instrument system 14 and/or the endoscopicimaging system 15.

The operator input system 16 may be located at a surgeon's console orother control console, which is usually located in the same room asoperating table O. It should be understood, however, that the surgeon Scan be located in a different room or a completely different buildingfrom the patient P. Operator input system 16 generally includes one ormore control device(s) for controlling the medical instrument system 14.The control device(s) may include one or more of any number of a varietyof input devices, such as hand grips, joysticks, trackballs, datagloves, trigger-guns, foot pedals, hand-operated controllers, voicerecognition devices, touch screens, body motion or presence sensors, andthe like. In some embodiments, the control device(s) will be providedwith the same degrees of freedom as the medical instruments of theteleoperational assembly to provide the surgeon with telepresence, theperception that the control device(s) are integral with the instrumentsso that the surgeon has a strong sense of directly controllinginstruments as if present at the surgical site. In other embodiments,the control device(s) may have more or fewer degrees of freedom than theassociated medical instruments and still provide the surgeon withtelepresence. In some embodiments, the control device(s) are manualinput devices which move with six degrees of freedom, and which may alsoinclude an actuatable handle for actuating instruments (for example, forclosing grasping jaw end effectors, applying an electrical potential toan electrode, delivering a medicinal treatment, and the like).

The teleoperational assembly 12 supports and manipulates the medicalinstrument system 14 while the surgeon S views the surgical site throughthe operator input system 16. An image of the surgical site can beobtained by the endoscopic imaging system 15, such as a stereoscopicendoscope, which can be manipulated by the teleoperational assembly 12to orient the endoscopic imaging system 15. An electronics system 18(also called an “electronics cart 18” when configured as a cart orconfigured to be usable as part of a cart) can be used to process theimages of the surgical site for subsequent display to the surgeon Sthrough the surgeon's console comprising the operator input system 16.The number of medical instrument systems 14 used at one time willgenerally depend on the diagnostic or surgical procedure and the spaceconstraints within the operating room among other factors. Theteleoperational assembly 12 may include a kinematic structure of one ormore non-servo controlled links (e.g., one or more links that may bemanually positioned and locked in place, generally referred to as aset-up structure) and a teleoperational manipulator. The teleoperationalassembly 12 includes a plurality of motors that drive inputs on themedical instrument system 14. These motors move in response to commandsfrom the control system (e.g., control system 20). The motors includedrive systems which when coupled to the medical instrument system 14 mayadvance the medical instrument into a naturally or surgically createdanatomical orifice. Other motorized drive systems may move the distalend of the medical instrument in multiple degrees of freedom, which mayinclude three degrees of linear motion (e.g., linear motion along the X,Y, Z Cartesian axes) and in three degrees of rotational motion (e.g.,rotation about the X, Y, Z Cartesian axes). Additionally, the motors canbe used to actuate an articulable end effector of the instrument forgrasping tissue in the jaws of a biopsy device or the like. Instruments14 may include end effectors having a single working member such as ascalpel, a blunt blade, an optical fiber, or an electrode. Other endeffectors may include, for example, forceps, graspers, scissors, or clipappliers.

The teleoperational medical system 10 also includes a control system 20.The control system 20 includes at least one memory and at least oneprocessor (not shown), and typically a plurality of processors, foreffecting control between the medical instrument system 14, the operatorinput system 16, and an electronics system 18. The control system 20also includes programmed instructions (e.g., a computer-readable mediumstoring the instructions) to implement some or all of the methodsdescribed in accordance with aspects disclosed herein. While controlsystem 20 is shown as a single block in the simplified schematic of FIG.1A, the system may include two or more data processing circuits with oneportion of the processing optionally being performed on or adjacent theteleoperational assembly 12, another portion of the processing beingperformed at the operator input system 16, and the like. Any of a widevariety of centralized or distributed data processing architectures maybe employed. Similarly, the programmed instructions may be implementedas a number of separate programs or subroutines, or they may beintegrated into a number of other aspects of the teleoperational systemsdescribed herein. In one embodiment, control system 20 supports wirelesscommunication protocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11,DECT, and Wireless Telemetry.

In some embodiments, control system 20 may include one or more servocontrollers that receive force and/or torque feedback from the medicalinstrument system 14. Responsive to the feedback, the servo controllerstransmit signals to the operator input system 16. The servocontroller(s) may also transmit signals instructing teleoperationalassembly 12 to move the medical instrument system(s) 14 and/orendoscopic imaging system 15 which extend into an internal surgical sitewithin the patient body via openings in the body. Any suitableconventional or specialized servo controller may be used. A servocontroller may be separate from, or integrated with, teleoperationalassembly 12. In some embodiments, the servo controller andteleoperational assembly are provided as part of a teleoperational armcart positioned adjacent to the patients body.

The electronics system 18 can be coupled with the endoscopic imagingsystem 15 and can include a processor to process captured images forsubsequent display, such as to a surgeon on the surgeon's console, or onanother suitable display located locally and/or remotely. For example,where a stereoscopic endoscope is used, the electronics system 18 canprocess the captured images to present the surgeon with coordinatedstereo images of the surgical site. Such coordination can includealignment between the opposing images and can include adjusting thestereo working distance of the stereoscopic endoscope. As anotherexample, image processing can include the use of previously determinedcamera calibration parameters to compensate for imaging errors of theimage capture device, such as optical aberrations. The electronicssystem 18 may also include a display monitor and components of thecontrol system 20.

The teleoperational medical system 10 may further include optionaloperation and support systems (not shown) such as illumination systems,steering control systems, irrigation systems, and/or suction systems. Inalternative embodiments, the teleoperational system may include morethan one teleoperational assembly and/or more than one operator inputsystem. The exact number of manipulator assemblies will depend on thesurgical procedure and the space constraints within the operating room,among other factors. The operator input systems may be collocated, orthey may be positioned in separate locations. Multiple operator inputsystems allow more than one operator to control one or more manipulatorassemblies in various combinations.

FIG. 1B is a perspective view of one embodiment of a teleoperationalassembly 12 which may be referred to as a patient side cart 12 whendesigned to be placed near the patient during a medical operation. Theteleoperational assembly 12 shown provides for the manipulation of threesurgical instruments 26 a, 26 b, 26 c (e.g., instrument systems 14) andan imaging instrument 28 (e.g., endoscopic imaging system 15), such as astereoscopic endoscope used for the capture of images of the site of theprocedure. The imaging device 28 may also be called “imaging device 28,”and may transmit signals over a cable 56 to the control system 20.Manipulation is provided by teleoperative mechanisms having a number ofjoints. The imaging device 28 and the surgical instruments 26 a-c can bepositioned and manipulated through incisions in the patient so that akinematic remote center is maintained at the incision to minimize thesize of the incision. Images of the surgical site can include images ofthe distal ends of the surgical tools 26 a-c when they are positionedwithin the field-of-view of the imaging device 28.

The patient side cart 22 includes a drivable base 58. The drivable base58 is connected to a telescoping column 57, which allows for adjustmentof the height of the arms 54. The arms 54 may include a rotating joint55 that both rotates and moves up and down. Each of the arms 54 may beconnected to an orienting platform 53. The orienting platform 53 may becapable of 360 degrees of rotation. The patient side cart 22 may alsoinclude a telescoping horizontal cantilever 52 for moving the orientingplatform 53 in a horizontal direction.

In the present example, each of the arms 54 connects to a manipulatorarm 51. The manipulator arms 51 may connect directly to a medicalinstrument 26. The manipulator arms 51 may be teleoperatable. In someexamples, the arms 54 connecting to the orienting platform are notteleoperatable. Rather, such arms 54 are positioned as desired beforethe surgeon S begins operation with the teleoperative components.

Endoscopic imaging systems (e.g., systems comprising imaging device 15or 28) may be provided in a variety of configurations including rigid orflexible endoscopes. Rigid endoscopes include a rigid tube housing arelay lens system for transmitting an image from a distal end to aproximal end of the endoscope. Flexible endoscopes transmit images usingone or more flexible optical fibers. Digital image based endoscopes havea “chip on the tip” design in which a distal digital sensor such as aone or more charge-coupled device (CCD) or a complementary metal oxidesemiconductor (CMOS) device store image data. Endoscopic imaging systemsmay provide two- or three-dimensional images to the viewer.Two-dimensional images may provide limited depth perception.Three-dimensional stereo endoscopic images may provide the viewer withmore accurate depth perception. Stereo endoscopic instruments employstereo cameras to capture stereo images of the patient anatomy. Anendoscopic instrument may be a fully sterilizable assembly with theendoscope cable, handle and shaft all rigidly coupled and hermeticallysealed,

FIG. 1C is a perspective view of a surgeon's console comprising theoperator input system 16. The surgeon's console comprising the operatorinput system 16 includes a left eye display 32 and a right eye display34 for presenting the surgeon S with a coordinated stereo view of thesurgical environment that enables depth perception. The consolecomprising the operator input system 16 further includes one or moreinput control devices 36, which in turn cause the teleoperationalassembly 12 to manipulate one or more instruments or the endoscopicimaging system. The input control devices 36 can provide the samedegrees of freedom as their associated instruments 14 to provide thesurgeon S with telepresence, or the perception that the input controldevices 36 are integral with the instruments 14 so that the surgeon hasa strong sense of directly controlling the instruments 14. To this end,position, force, and tactile feedback sensors (not shown) may beemployed to transmit position, force, and tactile sensations from theinstruments 14 back to the surgeon's hands through the input controldevices 36. Input control devices 37 are foot pedals that receive inputfrom a user's foot.

With reference to FIG. 2, each control device 36 includes a four degreeof freedom gimbal or wrist that allows rotation of an actuatable handle70 about three axes—axis A1, axis A2, and axis A3. The handle 70 iscoupled to a first elbow-shaped link 72 by a first rotatable joint 74that permits continuous rotation of the handle. The first link 72 iscoupled to a platform 76. Each control device 36 will generally allowmovement of the handle 70 within the workspace control workspace with aplurality of degrees of freedom, typically with six degrees of freedom,three rotational degrees of freedom and three translational degrees offreedom. This allows the actuatable handle 70 to be moved to anyposition and any orientation within its range of motion. The actuatablehandle 70 includes grip actuators 78 and/or switches 80 to allow theoperator to actuate the instrument 26 or 28 being positioned by themotion of the handle. Finger loops 82 are attached to the handle 70 toprevent the operator's fingers from slipping on the handle.

The surgeon's console comprising the operator input system 16 provides a“head-in” graphical user interface which allows the surgeon to perform avariety of functions with the medical instruments while his or her headremains in the console comprising the operator input system 16 with eyesviewing the displays 32, 34 and with hands remain engaged with thecontrol devices 36. As will be described below, an example of aprocedure that may be performed while the user's head remains in theconsole comprising the operator input system 16 is an instrumentexchange procedure. As features and interaction modalities of theteleoperational system and medical instruments increase, it becomesimpracticable to continue to add additional physical input controldevices (e.g., foot pedals, switches, buttons) for each additionalfeature. Including graphical menus in the user interface provide a rich,extensible platform to access instrument and system capabilities withoutadding additional physical user inputs.

FIG. 3 illustrates a surgical environment 100 within the patient P. Animaging instrument (e.g., imaging instrument 28) is used to display animage 102 of the imaging instrument's field of view within the surgicalenvironment 100. The image 102 may be a three dimensional image obtainedby a stereoscopic endoscope and generated as a composite image of theimages visible to a user through right and left eye displays. The fieldof view includes an image of a distal end portion of instrument 26 a, adistal end portion of instrument 26 b, and a distal end portion ofinstrument 26 c. The image 102 also includes information fields 104located at the periphery of the image which may include instructions tothe clinician, warnings, instrument identification information, statusinformation, or other information relevant to the surgical procedure.

In one embodiment, as shown in FIG. 3, accessing features andcapabilities for the instrument 26 b in the field of view is enabled bya graphical menu 110 superimposed on the image 102. The graphical menu110 is displayed proximate to image of the distal end of the medicalinstrument 26 b so that the surgeon can easily tell that the co-locatedgraphical menu 110 pertains to the medical instrument 26 b. In thisembodiment, a graphical menu 110 contains association informationrelated to the medical instrument 26 b, such as a numerical identifierof the teleoperational arm (e.g. Arm “3”) to which the instrument iscoupled. “Systems and Methods for Onscreen Identification of Instrumentsin a Teleoperational Medical System” are provided in U.S. ProvisionalApp. No. 62/134,297, filed Mar. 17, 2015, which is incorporated byreference herein in its entirety.

In some embodiments, a default location for placement of the graphicalmenu 110 is superimposed over a portion of the distal end of instrument26 b. In other embodiments, the graphical menu 110 is displayed adjacentto the distal end portion. The default location for the menu may dependupon the size of the distal end portion of the medical instrument 26 bin the field of view. For example, if the image is closely zoomed in andthe end effector occupies a large portion of the image, the menu may belocated on a proximal portion of one of the end effector jaws withoutobstructing the view at the distal end of the jaws. If, however, theimage is zoomed out and the end effector jaws are relatively small inthe image, the menu may be located on the joint region or the shaft toavoid obscuring the jaws. The placement of the menu 110 allows theclinician to access features of the instrument while remaining focusedon the surgical environment.

The graphical menu 110 has a minimized configuration in which functionsassociated with instrument 26 b are presented as small, minimizedfunction icons 112 a, 112 b radially arranged about the periphery of aselector icon 114. The selector icon 114 includes a pointer 116 and abody 118. In this embodiment, the numerical identifier for theteleoperational arm is located in a central region of the body 118.Responsive to movement of the control device 36 associated with theinstrument 26 b, the selector icon 114 is animated to pivot about thecenter of the body 118 to move the pointer 116 in a dial-like fashion.

As shown in FIG. 4, the graphical menu 110 may, optionally, include agrip indicator 120 that indicates the state of the end effector of theinstrument 26 b and a shaft indicator 122 that indicates the directionof the shaft of the instrument 26 b. If the instrument 26 b has atwo-grip end effector, the grips may be shown as an open grip indicator170 (FIG. 8A) or a closed grip indicator 172 (FIG. 8B). If theinstrument 26 b has a single blade end effector, the blade may be shownas a single finger 174 (FIG. 8C). The indicators 120, 122 assist theviewing surgeon by helping to understand which instrument the graphicalmenu is associated with. For example, where there is ambiguity about theassociation between menu and instrument, the surgeon may view theindicators 120, 122 and observe the configuration of the nearbyinstrument distal ends. The surgeon can then associate the nearbyinstrument with the end effector and shaft configurations that match theindicators 120.

FIG. 5 illustrates a method 150 for accessing features for the medicalinstrument 26 b or the coupled teleoperational arm 3 via the graphicalmenu 110. At a process 152, the field of view image 102 is displayedwhile the teleoperational system is in a surgical instrument control or“following” operational mode in which the movement of the instruments,including instrument 26 b follows the movements of the user's hand inthe associated control device 36. At a process 154, the surgeon causesthe teleoperational system to enter into a menu mode in which movementof the user's hand in the associated control device 36 is decoupled frommovement of the instrument 26 b and arm 3. In one embodiment, the menumode may be entered when the surgeon steps on a clutch pedal (e.g.,pedal comprising the input control device 37) of the console comprisingthe operator input system 16. After depressing the clutch pedal, controlof the medical instrument 26 b by the control device 36 is suspended.The control device 36 may move freely without corresponding movement ofthe medical instrument 26 b. In the menu mode, movement of the gripactuators 78 of the control device 36 is decoupled from the movement ofthe instrument 26 b end effector. The minimized graphical menu 110appears upon entry into the menu mode.

To provide greater information about each of the functions associatedwith indicators 120, 122, an expanded graphical menu 130 may bedisplayed, as shown in FIG. 6. Graphical menu 130 includes the selectoricon 114 and replaces the minimized function icons 112 a, 112 b withexpanded function icons 132 a, 132 b. In this embodiment, each of theexpanded function icons includes a pictogram 134 and a function label136 that describe the function associated with the function icon. Forexample, icon 132 a is associated with an “EXIT” function that returnsthe expanded menu 130 to the format of the minimized graphical menu 110.Icon 132 b is associated with an “EJECT” function which initiates aninstrument ejection sequence for the medical instrument 26 b. Theexpanded graphical menu 130 is displayed when the handle 70 of thecontrol device 36 is rotated about axis A1. Optionally, when the handle70 is rotated, a haptic force providing a “click” sensation, will beprovided to the surgeon through the control device 36 to provideconfirmation of the launch of the graphical menu 130.

The rotational movement of the handle 70 (about axis A1) used to openthe expanded graphical menu 130 or to move the selector icon 114 ismovement in a degree of freedom (i.e. the roll degree of freedom) thatis not used to operate the instruments when the system is in other modesof operation such as the surgical instrument control mode. Thus, thesurgeon may recognize that the roll degree of freedom about the axis ofthe control device is decoupled and used for menu selection, notinstrument operation. The orientation of the roll axis does not changein the menu mode (or generally when the clutch pedal is engaged) and ismaintained in alignment with the last orientation of the instrument endeffector. In other embodiments, the same degrees of freedom used tocontrol the selector icon may be used to control tool movement in thesurgical instrument control mode.

Returning to FIG. 5, at a process 156, one of the function icons 132 a,132 b may be identified from the menu 130. More specifically when theexpanded graphical menu 130 is first opened, the pointer 116 ispositioned in a neutral position such as upward, toward the icon 132 a.This neutral position of the pointer corresponds to a neutral positionof the control device. The neutral position of the control device may bethe most recent position of the control device in the instrument controlmode. In other embodiments, the pointer is directed in other radialdirections in the neutral position. In the embodiment of FIG. 6, whenthe handle 70 is rotated clockwise about axis A1, the pointer 116rotates clockwise from the neutral position toward the next functionicon 132 b. The handle 70 may be rotated, for example, less than 180° oreven less than 90° to cause the pointer 116 to move to next functionicon. In this example, the handle 70 is rotated approximately 40°. Ahaptic force, e.g., a “click” sensation, may be provided to the surgeonthrough the control device 36 to provide confirmation that the pointer116 is pointed at the next function icon 132 b in the radial sequence oficons.

After the clockwise turn of the handle 70 and the application of thecorresponding haptic force, the handle returns or “snaps back” to theneutral position, but the pointer 116 remains pointed at the functionicon 132 b. The return to neutral position allows minimal deviation ofthe pose of the surgeon's hand to provide for better ergonomics (e.g.,the surgeon's wrist is not required to turn excessively). Thisrecentering of the handle 70 also allows reentry into the instrumentcontrol mode to proceed more quickly. Although only two function iconsare shown, in other embodiments, function icons may be arranged aroundthe full circle of the selector icon. After each rotation of the handle70, the pointer advances to the next icon in the radial series, but thecontrol device 36 may return to the neutral position. The handle 70 mayalso be moved counter-clockwise about the axis A1 to cause the pointer116 to move counterclockwise.

The functions that are available for selection may be identified by agraphic characteristic of the icon. For example, disabled functions maybe indicated by color (e.g., a white pictogram and text), by a graphicfeature such as a box 188 (FIG. 9D), or by an “X” placed over thepictogram and/or text. Enabled functions may be indicated by colorand/or by corner brackets 186 of a predetermined size. (FIG. 9C)

When the pointer 116 is directed at the function icon 132 b, thatfunction icon is considered “selected.” The selected function may beindicated by a change in the appearance of the function icon 132 b. Forexample, the corner brackets 184 may have a narrower width as comparedto the corner brackets 186 in the enabled state. (FIG. 9B)

To activate the function associated with the icon 132 b, the surgeondepresses the grip actuators 78 or other buttons or switches at thecontrol console comprising the operator input system 16. In thisembodiment, the icon 132 b is associated with an EJECT function, anddepressing the grip actuators initiates the instrument ejectionsequence. When the selected function has been activated, the appearanceof the function icon 132 may change to indicate that the function isselected. For example, double corner brackets 182 may appear to indicateto the viewer that the EJECT function has been activated. (FIG. 9A)

The instrument ejection sequence may begin with a brief hold-off period.This period may be marked by a change in the graphical menu such as aprogressive change in color of the selector icon or the incrementalgrowth of a ring around the edge of the selector icon. Next, if theinstrument has a two-finger grip end effector, the grips open to releaseany tissue and the wrist of the instrument straightens into alignmentwith the instrument shaft. The shaft may retract slightly. During thesestages of the EJECT function, a progress indicator 138 on the selectoricon 114 indicates the current stage of the EJECT function. For example,the progress indicator 138 may be a graphic ring that incrementallychanges color, proceeding in a clockwise direction. Before the graphicring creates a full circle, the EJECT function may be cancelled by, forexample, opening the grips of the control device or releasing the clutchpedal. After the progress indicator 138 indicates that the instrument isstraightened, the instrument may be retracted in to the access cannulaalong the insertion axis of the cannula. During the retraction processthat brings the end effector into the cannula, the EJECT function maynot be interrupted.

The menu of FIG. 6 may be controlled by the surgeon's right hand in thecorresponding control device. For left hand controlled instruments, theicons may be arranged with an opposite configuration (i.e., the EJECTicon to the left of the END icon). In various embodiments, movement ofthe selector icon 114 may be coupled to the right or, alternatively, theleft hand control device. Different graphical user interface menus maybe associated with each hand so that the right hand selects from a firstmenu and the left hand selects from a second menu. The menu may,alternatively, be configured so that the menu items are selected bycounter-clockwise rotation. In other embodiments, the control device maymove in both clockwise and counterclockwise directions to select menuitems. In various alternative embodiments, more than one graphical menumay be co-located with the tips of the instrument in the image of thefield of view.

FIG. 7 illustrates a graphical menu 160 according to an alternativeembodiment. In this embodiment a selector icon 162 moves downward alonga linear path 163 in response to clockwise rotation of the controldevice handle 70 and upward in response to counterclockwise rotation ofthe control device handle. The function icons 164 are linearly alignedalong the linear path 163. The selection and activation processes forthis embodiment in which the icons are linearly aligned are generallythe same as described above when the icons are radially arranged.

FIG. 10 illustrates a graphical user interface 200 according to anotherembodiment of this disclosure. In this embodiment, a graphical menu 202may be associated with a global control mode selection rather than witha particular instrument or instrument drive. In this embodiment, themenu 202 is in a fixed location in the surgical view, centered over agraphical image 204 of an instrument navigator representing a componentof the teleoperational system. For example, the instrument navigator mayrepresent the configuration of the end effectors of the system. The menu202 appears when the surgeon enters an ADJUST-CONTROL mode bymanipulating a control device (e.g., a clutch foot pedal) on the controlconsole. The menu includes a selector icon 206, which in this embodimentincludes a partial circular shape with a pointer, and radially arrangedfunction icons 208, 210. In an initial position, the selector icon maybe pointed upright to indicate the current active control mode. Theselector icon 206 may be pivoted, as described, above by turning thecontrol handle 70 about the roll axis A1. When the handle is turnedapproximately 40° clockwise, the pointer of the selector icon 206 isdirected toward the function icon 208 to select a camera-controlfunction. As previously described, a haptic force may be provided togive the surgeon a “click” sensation when the selector icon reaches theicon 208. The function may be selected by squeezing the grip actuators78. Once in camera-control mode, the selector icon 206 may disappear.The camera-control mode may be exited when the surgeon releases the footpedal. As previously described, the handle 70 of the control device mayreturn to a neutral position after each click of the selector icon 206.FIG. 11 illustrates a modified graphical menu 212 that further includesan icon 214 for the function of relocate control. In this embodiment,the icon 214 is to the left of the icon 208 so the surgeon may rotatethe selector icon 206 counter-clockwise.

One or more elements in embodiments of the invention may be implementedin software to execute on a processor of a computer system such ascontrol processing system. When implemented in software, the elements ofthe embodiments of the invention are essentially the code segments toperform the necessary tasks. The program or code segments can be storedin a processor readable storage medium or device that may have beendownloaded by way of a computer data signal embodied in a carrier waveover a transmission medium or a communication link. The processorreadable storage device may include any medium that can storeinformation including an optical medium, semiconductor medium, andmagnetic medium. Processor readable storage device examples include anelectronic circuit; a semiconductor device, a semiconductor memorydevice, a read only memory (ROM), a flash memory, an erasableprogrammable read only memory (EPROM); a floppy diskette, a CD-ROM, anoptical disk, a hard disk, or other storage device. The code segmentsmay be downloaded via computer networks such as the Internet, Intranet,etc.

Note that the processes and displays presented may not inherently berelated to any particular computer or other apparatus. Variousgeneral-purpose systems may be used with programs in accordance with theteachings herein, or it may prove convenient to construct a morespecialized apparatus to perform the operations described. The requiredstructure for a variety of these systems will appear as elements in theclaims. In addition, the embodiments of the invention are not describedwith reference to any particular programming language. It will beappreciated that a variety of programming languages may be used toimplement the teachings of the invention as described herein.

While certain exemplary embodiments of the invention have been describedand shown in the accompanying drawings, it is to be understood that suchembodiments are merely illustrative of and not restrictive on the broadinvention, and that the embodiments of the invention not be limited tothe specific constructions and arrangements shown and described, sincevarious other modifications may occur to those ordinarily skilled in theart.

What is claimed is:
 1. A method comprising: displaying an image of afield of view of a surgical environment, wherein a first medicalinstrument in the field of view is coupled to a first manipulator in ateleoperational assembly; displaying a menu proximate to an image of thefirst medical instrument in the image of the field of view, the menuincluding a plurality of icons wherein each icon is associated with afunction for the first medical instrument; and identifying a selectedicon from the plurality of icons based upon a movement of an operatorcontrol device of a teleoperational operator control system.
 2. Themethod of claim 1 further comprising: receiving a command to disable theoperator control device from controlling operation of the first medicalinstrument.
 3. The method of claim 2 wherein the operator control systemcomprises a clutch mechanism and receiving the command to disable theoperator control device is caused by activation of the clutch mechanism.4. The method of claim 1 wherein displaying the menu includes displayinga selector icon in the field of view, wherein the selector icon ismovable in the field of view in response to movement of the operatorcontrol device to indicate one of the plurality of icons.
 5. The methodof claim 4 further comprising: moving the selector icon in the field ofview in response to rotation of the operator control device about a rollaxis.
 6. The method of claim 5 wherein, in an instrument control mode,the roll axis of the first medical instrument is an unused degree offreedom.
 7. The method of claim 1 further comprising: modifying theplurality of icons to display a plurality of modified icons in responseto rotation of the operator control device.
 8. The method of claim 1wherein displaying the menu includes displaying an end effector indictorindicating a configuration of an end effector of the first medicalinstrument.
 9. The method of claim 1 wherein one of the functionsrepresented by one of the plurality of icons is an eject function forthe first medical instrument from the surgical environment.
 10. Themethod of claim 1 further comprising: displaying a progress indicationfor the function associated with the selected icon.
 11. The method ofclaim 1 further comprising: receiving an activation trigger; andactivating the function of the first medical instrument associated withthe selected icon based on receipt of the activation trigger.
 12. Themethod of claim 11 wherein the activation trigger is activation of aswitch.
 13. The method of claim 11 wherein the activation trigger isclosing of a gripping mechanism of the operator control device.
 14. Themethod of claim 11 further comprising: initiating a hold-off periodbefore activating the function of the first medical instrument afterreceipt of the activation trigger.
 15. A non-transitory machine-readablemedium comprising a plurality of machine-readable instructions whichwhen executed by one or more processors associated with acomputer-assisted medical system device are adapted to cause the one ormore processors to perform a method comprising: displaying an image of afield of view of a surgical environment, wherein a first medicalinstrument in the field of view is coupled to a first manipulator in ateleoperational assembly; displaying a menu proximate to an image of thefirst medical instrument in the image of the field of view, the menuincluding a plurality of icons wherein each icon is associated with afunction for the first medical instrument; and identifying a selectedicon from the plurality of icons based upon a movement of an operatorcontrol device of a teleoperational operator control system.
 16. Thenon-transitory machine-readable medium of claim 15 wherein the methodfurther comprises: receiving a command to disable the operator controldevice from controlling operation of the first medical instrument. 17.The non-transitory machine-readable medium of claim 16 wherein theoperator control system comprises a clutch mechanism and receiving thecommand to disable the operator control device is caused by activationof the clutch mechanism.
 18. The non-transitory machine-readable mediumof claim 15 wherein displaying the menu includes displaying a selectoricon in the field of view, wherein the selector icon is movable in thein field of view in response to movement of the operator control deviceto indicate one of the plurality of icons.
 19. The non-transitorymachine-readable medium of claim 18 wherein the method furthercomprises: moving the selector icon in the field of view in response torotation of the operator control device about a roll axis.
 20. Thenon-transitory machine-readable medium of claim 19 wherein, in aninstrument control mode, the roll axis of the first medical instrumentis an unused degree of freedom.